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Abstract 

Ludwig Boltzmann had a hunch that irreversibility exhibited by a macroscopic system 
arises from the reversible dynamics of its microscopic constituents. He derived a nonlinear 
integro-differential equation - now called the Boltzmann equation - for the phase space density 
of the molecules of a dilute fluid. He showed that the Second law of thermodynamics emerges 
from Newton's equations of motion. However Boltzmann realized that stosszahlansatz, em- 
ployed in the derivation, smuggles in an element of stochasticity into the transport equation. 
He then proposed a fully stochastic description of entropy which laid the foundation for statis- 
tical mechanics. Recent developments, embodied in different fluctuation theorems, have shown 
that Boltzmann's hunch was, in essence, correct. 
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Everything existing in the Universe is the fruit of chance and necessity 

Diogenes Laertius IX 



1 Prologue 



BOLTZMANN transport equation has played an important role in basic and applied sciences. 
It is a nonlinear integro-differential equation for the phase space density of the molecules of 
a dilute gas. It remains today, an important theoretical technique for investigating non-equilibrium 
systems. It was derived by Ludwig Eduard Boltzmann (1844 - 1906) in his further studies on ther- 
mal equilibrium between gas molecules [1], published in the year 1872. Boltzmann did this work 
solely for purpose of addressing the conflict between time-reversal-invariant Newtonian mechanics 
and time-arrowed thermodynamics. Linear version of this equation [2] provides an exact descrip- 
tion of neutron transport in nuclear reactor core and shields. Linear transport equation constitutes 
the backbone of nuclear industry. It is indeed appropriate that the Indian Society for Radiation 
Physics (ISRP) has chosen Boltzmann transport equation as focal theme for the sixteenth National 
Symposium on Radiation Physics (NSRP-16), in Meenakshi College for Women, Chennai during 
January 18-21, 2006. The year 2006 marks the hundredth anniversary of Boltzmann's death. 

There are going to be several talks [3] in this symposium, covering various aspects of linear 
transport equation. However, in this opening talk, I shall deal with nonlinear transport equation. 

I shall tell you of Boltzmann's life-long struggle for comprehending the mysterious emergence 
of time asymmetric behaviour of a macroscopic object from the time symmetric behaviour of its 
microscopic constituents. In the synthesis of a macro from its micro, why and when does time 
reversal invariance break down? This is a question that haunted the scientists then, haunts us now 
and most assuredly shall haunt us in the future, near and far. 

The Second law is about macroscopic phenomena being invariably time asymmetric; it is about 
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macroscopic behaviour being almost always irreversible ^ Physicists think the Second law can 
not be derived from Newton's equations of motion. According to them, the Second law must be a 
consequence of our inability to keep track of a large number, typically of the order of 10^'^ or more, 
of molecules. In other words, the origin of the Second law is statistical. It is one thing if statistics 
is used merely as a convenient descriptor of a macroscopic phenomenon. It is quite another thing 
if we want to attribute an element of truth to such a description. Is it conceivable that nature is 
deterministic at micro level and stochastic at macro level? Can (microscopic) determinism give 
rise to (macroscopic) unpredictability? Boltzmann thought so. 

Boltzmann believed that the Second law is of dynamical origin. He proved it through his trans- 
port equation and H-theorem. At least he thought he did. Several of his fellow men thought 
otherwise. It is this fascinating story of the Second law that I am going to narrate to you in this talk. 
I am going to tell you of the insights that Boltzmann provided through his early work on transport 
equation and his later work that laid the foundation for Statistical Mechanics - a subject that aims to 
derive the macroscopic properties of matter from the properties of its microscopic constituents and 
their interactions. I am also going to tell you of nonlinear dynamics and chaos, subjects that have 
completely changed our views about determinism, dynamics and predictability. Now we know that 
determinism does not necessarily imply predictability. There are a large number of systems that 
exhibit chaotic behavior. Chaos and hence unpredictability is a characteristic of dynamics. Thus, 
Boltzmann's hunch was, in essence, right. It was just that he was ahead of his time. 

Boltzmann staunchly defended the atomistic view. He trusted atoms [4]. He was of the opinion 
that atomistic view helps at least comprehend thermal behaviour of dilute fluids. But the most 

^Deterioration, dissipation, decay and death characterize macroscopic objects and macroscopic phenomena. A 
piece of iron rusts; the reverse happens never. A tomato rots, inevitably, invariably and irreversibly. An omelet is easily 
made from an egg; never an egg from an omelet. 

The physicists are puzzled at the Second law. How does it arise ? An atom - the constituent of a macroscopic object, 
obeys Newton's laws. Newtonian dynamics is time reversal invariant. You can not tell the past from the future; there is 
the determinism - the present holding both, the entire past and the entire future. The atoms, individually obey the time 
reversal invariant Newtonian dynamics; however their collective behaviour breaks the time symmetry. 

The philosophers are aghast at the implications of the Second law. Does it hold good for the Creator ? They are 
upset at the Second law since it spoils the optimism and determinism imphcit in for example in the verse below from 
Bhagavat Gita, an ancient text from the Hindu Philosophy: 

Whatever happened, it happened 
for good. 
Whatever is happening, is 
happening for good. 
Whatever that will happen, it will 
be for good. 

Omar Khayyam surrenders to the irreversibiUty of life when he writes, 

The Moving Finger writes; and, having writ. 
Moves on: nor all your Piety nor Wit 
Shall lure it back to cancel half a Line, 
Nor all your Tears wash out a Word of it. 

Bernard Shaw, frustrated with the Second law, exclaims youth is wasted on the young. Mark Twain hopes fondly for 
Second law violation when he wonders life would be infinitely happier if only we could be bom at eighty and gradually 
approach eighteen. 
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influential and vociferous of the German-speaking physics community - the so-called energeticists, 
led by Ernst Mach (1838 - 1916) and Wilhelm Ostwald (1853 - 1932) did not approve of this. 
For them, energy was the only fundamental physical entity. They dismissed with contempt any 
attempt to describe energy or transformation of energy in more fundamental atomistic terms or 
mechanical pictures. This lack of recognition from the members of his own community allegedly 
led Boltzmann to commit suicide ^. Ironically, Boltzmann died at the dawn of the victory of the 
atomistic view. For, in the year 1905, Albert Einstein (1879 - 1955) established unambiguously the 
reality of atoms and molecules in his work [5] on Brownian motion. 

2 On the nature of things 

It all started with our efforts to understand the nature of matter, in general and of heat, in particular. 
Ancient man must have definitely speculated on the possibility of tiny, invisible and indivisible 
particles assembling in very large numbers into a visible continuum of solids and liquids and an 
invisible continuum of air that surround us. The Greeks had a name for the tiny particle: atom - the 
uncuttable. According to Leucippus (440 B.C.) and his student Democritus (370 B.C.) atom moves 
in void, unceasingly and changing course upon collision with another atom. Titus Lucretius Carus 
(99 B.C. - 55 B.C.) mused on the nature of things ^ . According to him all the phenomena we see 
around are caused by invisible atoms moving hither and thither ^. There was no role for God in his 
scheme of things. Atomism of the very early times was inherently and fiercely atheistic. Perhaps 
this explains why it lost favour and languished into oblivion for several centuries. 

3 Revival of Atomistic view 

The revival came with the arrival of Galileo Galilei (1564-1642) who wrote in the year 1638, of the 
air surrounding the earth and of its ability to stand thirty four feet of water in a vertical tube closed 
at the top with the open bottom end immersed in a vessel of water. He also knew of air expand- 
ing upon heating and invented a water-thermo-graph (thermometer). A few years later, his student 

^Boltzmann enjoyed the respect of all his colleagues. Rejection of his ideas by the energeticists does not seem to be 
the only reason or even one of the reasons that drove him to his tragic end. Men like myths. Men like heroes. Scientists 
are no exception. Scientists need heroes - tragic or otherwise. Boltzmann is one such. 

^Lucretius wrote a six books long poem called De Rerum Natum (On the Nature of Things) on atomism. He writes 

clothes hung above a surf- swept shore 
grow damp; spread in the sun they dry again. 
Yet it is not apparent to us how 
the moisture cUngs to the cloth, or flees the heat. 
Water, then, is dispersed in particles, 
atoms too small to be observable 

''The atoms are 

... shuffled and jumbled in many ways, in the course 
of endless time they are buffeted, driven along 
chancing upon all motions, combinations. 
At last they faU into such an arrangement 
as would create this universe.... 
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Evangelista Torricelli (1608-1647) correctly concluded of air pressure and surmised that mercury, 
fourteen times heavier, would rise in the tube only upto thirty inches. He showed it experimentally. 
Blaise Pascal (1623 -1662) was quick to point out that Torricelli's reasoning would imply that the 
pressure of air on top of a mountain should be less, which was also verified through experiments in 
1648. Daniel Gabriel Fahrenheit (1686 - 1736) invented the mercury thermometer and the Fahren- 
heit scale of temperature in the year 1714. Andres Celsius (1701-1744) invented the centigrade or 
Celsius scale of temperature in the year 1742. Robert Boyle (1627 - 1691) carried out numerous 
experiments on the static and kinetic nature of air pressure and showed that the product of pressure 
and volume of a given amount of air remains constant if the temperature is also kept constant. This 
is called Boyle's law ^. Boyle modeled air as a collection of springs that resist compression (which 
explains air - pressure) and expands into available space [6]. Guillaume Amontons (1663 - 1705) 
experimented on expansion of gases with increase of temperature under constant pressure. He pro- 
posed an absolute zero of temperature at which, volume of a gas becomes zero at constant pressure 
or the pressure becomes zero under constant volume. The absolute zero temperature calculated 
from Amontons' experiments turned out to be —248 ° C. But nobody took notice of the Amontons' 
suggestion of an absolute temperature scale and absolute zero of temperature ^. Another important 
work carried out in the early eighteenth century was that of Daniel Bernoulli (1700 - 1782), who 
gave a derivation of Boyle's law from his billiard ball atomic model [9]. Bernoulli's billiard ball 
atom moves freely in space, colliding with other billiard ball atoms and with the walls of the con- 
tainer. Bernoulli interpreted gas pressure as arising due to numerous impacts the billiard ball atoms 
make with the walls of the container. 



4 Caloric Theory 

Despite these remarkably insightful work, both experimental and theoretical, carried out in the 
seventeenth and early eighteenth century, kinetic theory did not take off. Scientists could not simply 
comprehend heat as arising out of atomic motion: be it undulating motion around fixed position, 
like Boyle imagined or free motion in the available space of the container, like Bernoulli modeled. 
This difficulty is perfectly understandable since it was known that heat could be transmitted through 
vacuum, like for example, the heat from the sun. Hence, heat can not be a property of a substance; 
it has to be a substance by itself. Antoine Lavoisier (1743 - 1794) gave the name Calorique (or in 
English Caloric) to this fluid substance. In fact the French chemists included Calorique as one of 
the elements in the list prepared in the late eighteenth century. Caloric fluid always flowed from 
higher to lower temperatures. Heat engines that produced locomotion from burning of coal started 
dotting the European country side. 

^Boyle got the idea from the paper of Richard Townley (1638 - 1707) describing the work Townley carried out with 
Henry Power, see S. G. Brush [8] Book 1; p.l2. 

*A century later, Jacques Alexandre Cesar Charles (1746 - 1823) and Joseph Louis Gay-Lussac (1778 - 1850) 
estabUshed the law of thermal expansion of gases as we know of it today: The pressure (at constant volume) or 
the volume (at constant pressure) is proportional to T -|- a, where T is the temperature measured in some scale say 
Fahrenheit or Celsius; a is a constant that depends on the scale chosen for T. We can define T + a as absolute 
temperature whose zero will be lowest attainable temperature; in the scale chosen for measuring T the lowest attainable 
temperature is thus —a. In fact the notion of absolute scale and absolute zero of temperature got the acceptance of the 
scientific community only after William Thomson (Kelvin) proposed it [20] in the year 1848 based on Carnot engine 
i.e. the Second law: the partial derivative of entropy with respect to energy gives the inverse of absolute temperature. 
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5 Carnot's Engine and Caloric heat 

Nicolas Leonard Sadi Camot (1796 - 1832) was investigating why a French heat engine delivered 
invariably less work than its British counterpart. Carnot was intrigued by the very idea of a heat 
engine which manages to do what even the almighty Nature could not: A heat engine converts heat 
into movement. In nature you find that it is the movement which due to friction generates heat 
and never the the other way. There is no phenomenon like un-friction or anti-friction which would 
spontaneously re-assemble the heat back into a movement. Thinking along these lines Camot came 
to the conclusion [10] that mere production of heat is not sufficient to give birth to the impelling 
power; it is necessary there should be cold; without it, heat is useless. Thus the work produced 
should depend on the temperature difference between the boiler (heat source) and the radiator (the 
heat sink). This is a remarkable finding. The heat engine is like a mill wheel. A mill wheel simply 
extracts work from falling water. Larger the quantity of water and higher the fall, more is the work 
produced in the mill wheel. Analogously, larger the heat source and higher the temperature fall, 
more is the work produced in the heat engine. If a certain quantity q of caloric falls from absolute 
temperature Ti to zero, then the work produced will ht W = q; since it falls only to a finite 
temperature 72 (0 < 2^2 < Ti), only the proportional fraction of q should equal the work produced. 
In the year 1824, Camot announced in his historic treatise [10] entitled. Reflexions on the motive 
power of fire and on machines to develop that power, that the ratio of work (W) delivered by a heat 
engine to the heat (q) generated in the boiler at temperature Ti, is given by 

W T — T 

?7 = — = ' ' < 1 for < ^2 < < oo , (1) 
Q J-i 

where T2 is the temperature of the heat sink (the radiator). Even ideally, a heat engine can not have 
unit efficiency. The best you can get is Carnot's efficiency given by Eq. (1). When Camot measured 
the actual work delivered by a heat engine it was much less than what his formula suggested. Real- 
life heat engines have moving parts that mbbed against each other and against other parts; the 
resulting friction - which produces heat from work - is thus completely antagonistic to the heat 
engine which is trying to produce work from heat. Not surprisingly a practical engine is less 
efficient than Camot's ideal engine. In fact Carnot's engine is a double idealization: its efficiency 
is less that unity since it is not realistic to set T2 to zero; it should also work without friction which 
is not practical either. 

Camot's picture of a heat engine is completely consistent with the Caloric theory of heat. In fact 
it constitutes a triumph of the Caloric theory. Water that rotates the mill wheel is never consumed. 
Likewise the Caloric fluid that powers the heat engine is never destroyed. In the radiator the Caloric 
fluid is reabsorbed in the water and returned to the boiler for conversion to steam again. 

It looked like the Caloric theory had come to stay for good. It was becoming immensely and 
increasingly difficult for the kinetic heat to dethrone the Caloric heat and regain its lost and for- 
gotten glory. A sense of complacency started prevailing amongst the scientists at that time. There 
arose a certain reluctance to accept new ideas. It often happens in science: when a scientific theory 
is confirmed and firmly established, it loses its character and becomes a dogma; the practitioners 
of the theory become dogmatic. 
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6 The tragedy of Herpath and Waterston 

Consider the manuscript of John Herpath (1790-1868) submitted in the year 1820, containing new 
ideas on kinetic theory of heat. Herpath, unaware of Bernoulli's work, proposed an atomic model 
for the gas; he said heat is proportional to total momentum of the molecules of gas and absolute 
temperature corresponds to momentum per gas molecule. Herpath's work was found to be too 
speculative. The Royal Society did not find it fit to publish it in their Philosophical Transactions. 
Obviously the reviewers were also unaware of the work of Bernoulli. 

The same fate awaited the brilliant work scripted by John James Waterston (1811 - 1883), then 
at Bombay (now called Mumbai) and submitted in the year 1845, to the Royal Society. Waterston's 
model of gas contained molecules moving incessantly and colliding with each other and with the 
walls of the container. Waterston correctly identified the temperature as measuring the energy of 
motion of the molecules. One of the two reviewers considered Waterston's work as ' nothing but 
nonsense ' . The other reviewer was less harsh. He wrote that Waterston's suggestion that the 
pressure is due to molecular impacts on the walls of the container was ' extremely hypothetical and 
difficult to admit ' . The manuscript was rejected and buried in the archives of the Royal Society. 
Much later, in the year 1891, John WiUiam Strutt (Lord) Rayleigh (1842 - 1919) stumbled on 
the 1845-manuscript of Waterston; to his astonishment he found it contained essentially the same 
ideas proposed by August Karl Kronig (1822 - 1879) in 1856 [12] and by Rudolf Julius Emmanuel 
Clausius (1822 - 1888) in the year 1857 and in the later years. He got Waterston's manuscript 
published [13] in the Philosophical Transactions of the Royal Society, in the year 1893. 

7 Experimental evidence against Caloric heat 

But then there were significant developments in experimental thermal physics that started knocking 
at the very foundations of the Caloric theory. 

In the year 1798, Benjamin Thompson Rumford (1753 - 1814) noticed [14] that a canon became 
hot while boring. The heat it generated was sufficient to melt the canon. This means that the 
Caloric fluid produced is more than what was originally contained in the canon. This is not possible 
under Caloric theory. Julius Robert von Mayer (1814 - 1878), in the year 1840, came to the same 
conclusion [15] that heat is like mechanical energy. The paddle wheel experiment of James Prescott 
Joule (1818 - 1889) [16] carried out in the year 1845 established the mechanical equivalence of 
heat ^. These experiments of Rumford, Mayer and Joule, thus established unambiguously that the 
Caloric theory of heat was wrong and heat, like work, is actually energy or more precisely energy 
in transit, see section 15.1. Once we identify heat with energy, Carnot's finding becomes intriguing. 
Why? 

The first law of thermodynamics ^ tells us, energy can neither be created nor destroyed. How- 
ever energy can be converted from one form to the other. Carnot's finding amounts to saying that 
heat energy can not be converted completely into mechanical energy whereas mechanical energy 

^1 Calorie = 4.184 Joules where Joule is the SI unit of energy denoted by the symbol J and given by, 1 J=lKg. 
M^/sec^. 

^This principle of conservation of energy, called the first law of thermodynamics, was proposed independently by 
several scientists in the middle of the nineteenth century, notable amongst them are Mayer [15], Joule [16, 17] and 
Heknholtz [18]. 
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can be completely converted into heat ^ There is a kind of thermodynamic irreversibility. In the 
(first-) law abiding democratic society of energies, heat occupies a special place. Perhaps it is like 
what Bernard Shaw said: In a democracy, all men are equal but some are more equal than others. 
There is an apparent injustice in nature's scheme. 

Nobody took notice of Camot's work for over two decades. Benoit Paul Emilie Clapeyron 
(1799 - 1864) felt that Camot had discovered something profound. He provided the required phys- 
ical and mathematical scaffolding [19] which caught the attention of William Thomson (Kelvin) 
(1824 - 1907) and Clausius. Kelvin proposed [20] an absolute temperature scale based on Camot's 
engine. 

8 Clausius invents Entropy 

Clausius was intrigued by Carnot's finding. He felt that Camot's basic conclusion is correct and 
also considered it as of great fundamental importance. He called it the Second law of thermo- 
dynamics. But then he rejected Camot's reasoning based on Caloric theory of heat. From the 
experiments of Rumford, Mayer and Joule, he understood that heat and work are simply two dif- 
ferent forms of energy transfer, see section 15.1. He had known by then that heat was a kind of 
motion [21]. To explain Camot's finding in the context of this emerging picture, Clausius invented 
a new thermodynamic variable. His reasoning was simple. 

Consider a thermodynamic process described by a path in an appropriate phase space of ther- 
modynamic variables like internal energy {E), volume (V), pressure (P), temperature (T), number 
of molecules {N), chemical potential (/i) etc. During the process, the system absorbs or liberates 
energy in the form of heat (Q) and/or work {W). Both Q and W are path-dependent. Hence 
they are not state variables. In other words dQ and cf are not perfect differentials However 
dW — PdV. Inverse of pressure provides integrating factor for work. Clausius discovered that 
inverse of temperature provides integrating factor for heat. The quantity dQ/T tumed out to be 
a perfect differential. Clausius denoted this perfect differential by the symbol dS. There was no 
known thermodynamic state variable, whose perfect differential corresponded dS. Clausius, in his 
1865 paper [22], named the state variable SmdS = dq/T as entropy Let me quickly illustrate 
this on a simple example. 

^Ginsberg's restatement of the three laws of thermodynamics: 

First law : You can't win; 
Second law : You can't even break even; 
Third law : You can't even quit. 

^"The isotherms and the adiabats in the pressure - volume phase diagram (describing Camot's engine) that you find 
in text books on thermodynamics were actually drawn by Clapeyron 
''The cross on d denotes they are not perfect differentials. 

'^in the words of Clausius ... We now seek an appropriate name for S We would call S the transformation 

content of the body. However I have feU it more suitable to lake names of important scientific quantities from the 
ancient languages in order that they may appear unchanged in all contemporary languages. Hence I propose that we 
call S the entropy of the body after Greek word ' rjTpoiTr], meaning " transformation " . I have intentionally formed 
word entropy to be as similar as possible to the word energy, since the two quantities that are given these names are 
so closely related in their physical significance that a certain likeness in their names has seemed appropriate. 
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Start with the first law of thermodynamics, 

dU = dQ + dW 



(2) 



Consider an ideal gas to which energy in the form heat is supplied at constant volume; its internal 
energy increases by dU = CydT, where Cy is the specific heat at constant volume. The ideal gas 
law is given by PV = QT, where 6 is a constant. From this we get 



dv^e 

The work done (—PdV — 0) is given by 
Therefore we have, 



1 T ' 



0. 



T 

-dP-dT 



= 0. 



T 



dQ = (Cv + e) dT -e -dP . 
Let us investigate if dQ is a perfect differential. From the above, we have 

- r +B 



Differentiating once more we get. 



dQ 
dP 



d'Q 
dPdT 

d'Q 
dTdP 



-e 



T 



e 
p' 



Therefore, 



d-'Q , d^Q 



dPdT ^ dTdP ' 



(3) 



(4) 
(5) 

(6) 
(7) 



(8) 
(9) 

(10) 



(11) 



showing that dQ is not a perfect differential and Q is not a state function of P and T. We shall 
cross the ' to denote this. 

Consider now, the quantity dS — dQ/T, obtained by dividing all the terms in Eq. (5) by T. 
We have. 



dS_ 
&T 

dS_ 
dP 



Cy + Q 

T 

_e 
p ' 



(12) 
(13) 
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It is easily seen that 



dPdT dTdP 



= , (14) 



demonstrating that S* is a state function. Clausius gave the name entropy to this state function . 
Thus for the very definition of entropy, we need a thermodynamic process that can be represented 
by a path in the space of state variables. We call this a quasi-static process, described below. 

8.1 Quasi-static processes 

Consider a macroscopic system in equilibrium e.g. a gas in a cylinder fitted with a piston. Let 
A be a degree of freedom which can be manipulated from outside. For example A can be taken as 
volume of the gas which can be changed by moving the piston. Consider a thermodynamic process 
in which we switch the value of A from say Aq to A,- over a duration of time r. This switching can 
be carried out with some pre-determined protocol. For example we can change A uniformly. We 
say the process becomes quasi-static when the switching takes place extremely slowly. Strictly for 
a quasi-static process r equals infinity. It is a process of slow stepping through equilibrium states. 
At each step the state variables assume the relationship given by equation of states; the system 
is sort of dragged through a set of dense succession of equilibrium states. A quasi-static process 
can not be realized in practice. At best we can approximate it by an extremely slow process. A 
quasi-static process is reversible if it takes place at constant total entropy. In other words during 
a quasi-static process the change in entropy of the system plus the change in the entropy of the 
surroundings equals zero. 

8.2 The Second law: dS > 

For defining entropy, Clausius considers a quasi-static reversible process. During the process 
the system absorbs a quantity cf Qrev of reversible heat, from a heat source at temperature T. The 
entropy of the system increases by an amount given by, 

dS = (15) 

Since the process is quasi-static and reversible, the entropy of the heat source decreases by precisely 
the same amount so that the total change in entropy is zero 

Consider an isolated system and let dS denote the change in entropy of the system during a 
process. If the system is not isolated, then ds denotes the change in entropy of the system plus the 
change in entropy of its surroundings. Clausius states the Second law as, 

dS > 0, (16) 

in any thermodynamic process. In the above, equality obtains when the process is quasi-static and 
reversible. With this Second law assertion, Clausius was able to show that the efficiency of any 
heat engine is less than or equal to that of Carnot's engine, see below. 



'^For the definition of entropy the reversibility of the quasi-static process is only a sufficient condition but not 
necessary; the necessary condition is that the process should be quasi-static. 
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9 Kinetic heat replaces Caloric heat in Carnot's engine 

Consider an engine, M, which, operating in a cycle, draws a quantity qi of energy in the form 
of heat quasi-statically and reversibly from a heat reservoir (R) at temperature Ti. Let us say the 
engine converts the entire heat qi into work W and returns to its initial state. Such an engine is 
called a perfect engine, shown in Fig. (1). Under the first law of thermodynamics it is possible, 
in principle, to construct a perfect engine. Let us investigate what happens when we impose the 
Second law. The change in entropy of the heat source is — gi/Ti. Since the engine returns to its 
initial thermodynamic state there is no change in its entropy. We just saw that entropy is a state 
function. Thus, the total change in the entropy is dS — —qi/Ti. The Second law demands that 
dS > 0. The machine can not deliver work. Second law forbids perfect engines. However, the 
engine can convert mechanical energy W completely into heat, since during such a process dS > 0. 

Consider now an ideal engine M, shown in Fig. (2). It draws a quantity qi of energy in the form 
of heat, quasi-statically and reversibly from a source, R, kept at temperature Ti, it converts a part 
of it into work; it junks the remaining part q2 < qi into a sink (S), kept at temperature T2 < Ti; 
then it returns to the state it started with. 




Fig 1: Perfect Engine 



From the first law we have qi — q2 = W . The efficiency of the engine is given by, t] = W/ qi = 
1 — {q2/qi)- The change in entropy of the heat source is — gi/Ti and that of the sink is ^2/22- Since 
the machine returns to its initial state its entropy does not change. Therefore we have, 

dS^^-^. (17) 
T2 Ti 

The Second law demands that ^5" > 0. For an ideal engine dS = 0. Therefore, for an ideal engine 
Q2/qi = T2/T1, from which we get 77 = 1 — (T2/T1), a result identical to what Carnot obtained for 
his Caloric fluid, see Eq. (1). 
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Fig 2: Ideal Engine 



This is precisely how the Caloric heat became kinetic heat in Camot's engine. If heat is kinetic 
i.e. motion, then what kind of motion is it? It must be the irregular motions of the gas molecules; 
for, the regular motion is already described by work given in terms of pressure and change of 
volume. Entropy, which is heat divided by temperature must be a measure of this irregularity of 
molecular motions; a measure of disorder; a measure of randomness. James Clerk Maxwell (1831 
- 1879) asserted that the very Second law that talks of increasing entropy, must be statistical in 
character; hence it should be possible to contravene the Second law with non zero probability. 
He even proposed a demon - now called Maxwell's demon - that violates the Second law For 
Maxwell, stochasticity was intrinsic to macroscopic behaviour 



'"^For an interesting account of Maxwell's demon and other demons, see [24]. 

'^Maxwell was amongst the first to recognize the need for statistical approach to kinetic theory. In fact his derivation 
of the distribution of speed of the molecules of an ideal gas is ingenious and elegant, see [25]. He assumes that the 
three components {vi, V2, v^), of the velocity 77 of an ideal gas molecule are independent and identically distributed: 
f{vi,V2, V3) = f{v\)f(v2)f{v2,), where / is the density of molecules in the velocity space. He argues that since there 
is no preferred direction of motion the function / must depend only on vl+v^+v^; this leads to functional equation: 
/('^i)/(^2)/(^^3) = (l>{vi + V2 + wf) whose solution is the famous MaxweU-Boltzmann distribution of molecular 
speed, 



f{v) = 4w 



2TTkBT\ ^^\^2 

m J 



V exp 



mv 
2kBT 



that we are all familiar with. In the same paper [25], Maxwell correctly recognizes that the Maxwell-Boltzmann distri- 
bution is a simple consequence of Central Limit Theorem concerned with additive random variables, see footnote (30). 
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10 Boltzmann Transport Equation 

However Boltzmann, at least in his early years, felt there was no need to invoke statistics to com- 
prehend the Second law. At the age of twenty two, Boltzmann wrote on the mechanical meaning 
of the Second law of the theory of heat [23]. Of interest to us is Boltzmann's paper [1] published 
in the year 1872 in which he derived his transport equation and in which he also announced the 
H theorem to prove the Second law. Boltzmann considers the density /(r, t) of molecules, each 
of mass m, at the six-dimensional phase space point (r, p), and at time t. Aim is to find an 
equation (of motion) for this function. The density changes with time since molecules enter and 
leave a given six dimensional phase space volume element d^rd^p at the phase space point (r, p). 
Let F denote an external force {e.g. due to gravitation) acting on the molecules. Suppose there 
are no collisions. A molecule at (r, p) at time t will be found at {f + p At/irh,p + FAt) at time 
t + At. Hamiltonian evolution preserves volume element d^rd^p along a trajectory, called Liouville 
theorem Therefore, 

At 

f(r + p —,p + FAt,t + At) = f{r,p,t). (18) 
m 

When there are collisions, we must add the contribution from collisions and write 

f{f+p^,p + FAt,t + At) = f{f,p,t)+(^] At. (19) 



Col. 



Taylor-expanding to first order in At and taking the limit At — > we get. 



| = -ip.V./-F.V,/+(f) , (20) 

dt m ^ \dtJcoi. 

where Vr and Vp are the gradient operators with respect to position and momentum, respectively. 
Boltzmann proposes a simple model for the collision term, see below. 

Consider only binary collisions, true for a dilute gas, where a pair of molecules with momenta 
Pi and p2 bounce off, after a collision, with momenta p'l and p'2, respectively. Let /(r, pi, P2, t) 
denote the density of pairs of particles with momenta pi and p2 at position r and at time t. 

10.1 Stosszahlansatz 



Boltzmann invokes stosszahlansatz - collision number assumption - of Maxwell, which states, 

f{'r,Pi,P2,t) = f(r,Pi,t) f{f,p2,t). (21) 

The above is also called the assumption of molecular chaos. The momenta of two particles are 
uncorrelated. The stosszahlansatz is time symmetric. For both Maxwell and Boltzmann, this as- 
sumption looked innocuous and self evident. From this, Boltzmann derives an expression for the 
collision term, as described below. 

'''Classically a particle is specified by three position and three momentum coordinates. It is represented by a point 
in the six-dimensional phase space, called the jj. space. A system of N particles is represented by a point in a 6N 
dimensional phase space called T space. 

'^discovered by Joseph Liouville (1809-1882). 

'^Eq. (19) can be taken as definition of the colUsion term. 
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Let (Ppi, (Pp2, d^Pi and d^p2 be the momentum volume elements at pi, p2, p'l, and p'2 re- 
spectively. Let us consider binary collisions that knock a molecule from d^pi into d^p[, while its 
collision partner gets knocked from d'^p2 into d^p2. Since we are interested only in the collision 
term, we shall omit, for notational convenience, reference to the dependence on position r and time 
t. The rate at which these collisions take place is given by. 



f{Pi)d Pif{p2)d P2 ^{puP2,p'i,p'2)d p'ld P2. 

In the above, S denotes the rate of transition from (pi,p2) to {p'i,P2)- The total rate of binary 
collisions that result in molecules getting knocked out of volume element d^pi is given by. 



R{0\JT)^d'pJ{pi) I d'p2 I dVl / d'p'2 f{p2)^Pl,P2,p\,p'2)- 



(22) 



While carrying out the integrals in the above, we must ensure that momentum and energy are con- 
served. Let i?(IN) denote the rate of binary collisions that knock molecules into the volume element 
d^Pi. This can be obtained exactly the same way described above except that we interchange the 

— * — * 

labels of momenta before and after collision: pi ^ p\ and p2 ^ p'2- In other words we consider 
binary collisions that knock molecules from d^p[ into d^pi and from d^P2 into d^p2. We get. 



Rim) = d'p, I d'p2 I d% I d% f{p\)f{p'2) m,P'2,Pl,P2). 



3^' 



(23) 



We consider molecule - molecule interaction potential to be spherically symmetric. We first note 
that a binary collision is time symmetric . In other words, the process seen in reverse is also an 
acceptable collision process. Hence, 



S(pi,P2,P'l,P'2) = ^{-P'l, -p'2, -Pl, -n)- 

Also E is unchanged under simultaneous reflection of all momenta: 

S(P1,P2,P'1,P'2) = -P2, -p'l, -^'2)- 

Combining the above two we get, 

S(pi,P2,P'l,P'2) = ^{P'1,P'2,P1,P2)- 

Thus we can write the collision term as. 



dt 



Col. 



d'p2 / ciVl / d^p'2 E(pi,P2,P'l,P'2) 



/(P'l)/(P'2) - /(Pl)/(P2) 



(24) 



(25) 



(26) 



(27) 



E depends on the geometry of collision, the relative velocity of the two particles entering collision 
and the nature of the colliding particles. The full nonlinear Boltzmann transport equation reads as. 



dt 



1 



m 



Pi.V./-F.Vp/ + 



j d'p2 J d%J d'p'2 E(pi,p2,p'l,p''2) 



f {p'l) f {p'2) - /(Pl)/(P2) 



(28) 
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11 Boltzmann H function 



Boltzmann defines his famous H function. 



H{t) = J d?p /(p,t)log[/(p,t)], 



(29) 



and then shows that a density /(p, t) that solves the transport equation obeys, 

dH 



dt 



< 0. 



(30) 



The above is clearly time asymmetric. In contrast to Newtonian dynamics which does not distin- 
guish the future from the past, the //-function has a well defined direction of time, which is what 
the Second law is all about. To prove the H theorem, we write from Eq. (29), 



Therefore, 



f ^/All + log(/)l|. 



-— — U implies — = U. 
dt ^ dt 



(31) 



(32) 



The H function does not change with time when the system is in equilibrium. Eq. (31) in con- 
junction with the transport equation, see Eq. (28) yields after a few simple steps, the following 
expression for the time evolution of the H function. 



dH 
'dt 



d'pi / d'p2 / d% / d% E(pi,pl,p'i,p'2) 



/(Pl)/(P2) - /(P'l)/(P'2) 



X 



I0g[/(P1)/(P2)] - l0g[/(p'i)/(p'2) 



We recognize that due to the concavity of the logarithm function, 

(y — x) (log y — log x) > y x,y > 



(33) 



(34) 



H decreases with time monotonically giving rise to an arrow of time for macroscopic evolution. 
Thus Boltzmann, like a magician, produced a time asymmetric rabbit from a time symmetric hat! 
The crucial point overlooked was in the usage of the stosszahlansatz before and after collision. 

— * — * 

Momentum conservation, pi+P2 — p'l + p'2> tells us that writing. 



/(P'1,P'2)=/(P'1)X/(P'2), 



(35) 



is not correct, since a pair of uncorrected particles gets correlated after collision. The reversibility 
paradox [26] of Josef Loschmidt (1821 - 1895) and the recurrence paradox [27] of Ernst Zer- 
melo (1871 - 1956) showed Boltzmann's claim was untenable. Let me quickly tell what these two 
paradoxes are. 
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11.1 Loschmidt and reversibility paradox 

Loschmidt's argument was based on microscopic reversibility. Consider an isolated system that 
evolves from time t = to time t = r. Let there be a spontaneous increase of entropy during 
this evolution. At time t — t reverse the momenta of all the molecules. Allow the system to 
evolve from time t = r to time t = 2r. At time t = 2t reverse once again the momenta of all 
the molecules. Since the system obeys time-reversal invariant Newtonian dynamics, it will end up 
at the same phase space point it started from. There would be a decrease in entropy during the 
evolution from time t = t to time t = 2r, contrary to the claim made by Boltzmann. This is called 
Loschmidt's reversibility paradox 

11.2 Zermelo and Recurrence paradox 

Zermelo argued that an isolated system, under Hamiltonian dynamics will return arbitrarily 
close to its initial point in the phase space and infinitely often. This is called recurrence theorem, 
discovered by Poincare [28, 29]. According to Poincare recurrence theorem, every dynamical sys- 
tem is at least quasi-periodic if not exactly periodic. This follows from Liouville theorem: a phase 
space volume of initial conditions evolve without change of its volume. Hence it is described by 
a tube shaped region of ever-increasing length. As the total region of phase space available to the 
dynamical system is finite, the tube must somewhere intersect itself. This means that the initial and 
final states eventually come close to each other. The dynamical system returns arbitrarily close to 
its initial state and it does so infinitely often. If there is a spontaneous increase of entropy during 
an interval of time, there will be a spontaneous decrease of entropy during the interval of Poincare 
recurrence; this contradicts Boltzmann's claim 



12 Statistical Entropy of Boltzmann 

Boltzmann conceded that perhaps, the use of stosszahlansatz has smuggled in an element of stochas- 
ticity (albeit in a very subtle way) into his otherwise purely dynamical derivation of the transport 
equation. He contended correctly that his H theorem is violated only when the system starts off 
from some special microstates which are very small in number. For an overwhelmingly large num- 
ber of initial conditions, the dynamical evolution does obey the H theorem. In other words, the 
typical behaviour of a macroscopic system is invariably consistent with the H theorem. 

Nevertheless, in the year 1877, Boltzmann changed tack completely and proposed a fully 
stochastic approach to the problem of macroscopic irreversibility. He presented his ideas in a 
paper [30] on the relation between the Second law of thermodynamics and probability theory with 

'^Time reversal as discussed in the text can be implemented in a computer employing molecular dynamics simulation 
techniques. We find that even small errors in the calculations of positions and momenta of the molecules are sufficient 
to reduce and eventually eliminate this effect. The phase space trajectory of the macroscopic system is extremely 
unstable with respect to initial conditions. Two arbitrarily close trajectories move arbitrarily far apart asymptotically. 
This is called chaos. This was known to Julius Henry Poincare (1854 - 1912) [28, 29], a contemporary of Ludwig 
Boltzmann. Chaos contains the seed for modem developments in non-equiUbrium statistical mechanics. We shall see 
more on these issues later. 

^"Poincare recurrence is easily observed in systems with a very few degrees of freedom. But the recurrence time 
increases exponentially with the system size i.e. with the number of molecules. Hence Poincare recurrence is seldom 
observed in the thermodynamic hmit. 
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respect to the law of thermal equilibrium. Of course Boltzmann interprets probability in a dynam- 
ical way: The probability of finding a system in a region of its phase space is the fraction of the 
observation time the dynamical trajectory spends in that region. 

Consider an isolated macroscopic system of N particles. It is represented by a point in a 6A^ 
dimensional phase space (F space), moving incessantly along a trajectory dictated by its dynamics. 
Let us coarse-grain the phase space in terms of hyper cubes each of volume h^^ . It is like a graph 
sheet that coarse-grains a plane in terms of tiny squares. Here h represents a constant having the 
dimension of action . A phase space hyper cube is called a microstate. Let x be the 6A^ dimen- 
sional vector denoting the phase space point of the system and let p{x, t)d^^x be the probability of 
finding the system in an infinitesimal volume (f^x at x at time t. Let the system be in equilibrium. 
In other words the density p is independent of time. Let {pi\ denote the discrete representation of 
the phase space density p{x). 

Boltzmann's H function, see Eq. (29), is then given by, 

// = ^Alog(A) (36) 

i=l 

where Q. is the total number of microstates accessible to the system under macroscopic constraints 
of energy U, volume V and number of molecules A^. Boltzmann defines entropy as, 

n 

S{U, V, N) = -kB Pi log(A) , (37) 
1=1 

where /cs is now called the Boltzmann constant 

If we assume that all the microstates are equally probable, then pi — 1/0 V i, and we get the 
famous formula for Boltzmann entropy, 

S^kB\og{h), (38) 

engraved on his tomb in Zentralfriedhof, Vienna Notice Boltzmann defines absolute entropy. In 
thermodynamics only change in entropy is defined. 

12.1 Is Boltzmann entropy consistent with thermodynamic entropy? 

Let V be the number of coarse-grained volume cells occupied by N non interacting molecules. 
For simplicity we ignore the momentum coordinates. Number of ways of configuring N molecules 
in V cells is given by = , from which it follows ^'^ S = ksN \og{y). Pressure is temperature 
times the partial derivative of entropy with respect to volume. We have, 

(39) 

BV V T' ' 



^'Now we identify h with Planck's constant; h = 6.626 x 10~^^ Joules-second. 

^^ks = 1.381 X lO^^'' Joules per degree kelvin. 
Strangely, Boltzmann never wrote down this formula in any of his papers, though he implied it. It was Max Planck 
who wrote it down explicitly from the H function. 

^"^This expression for entropy is not extensive - called Gibbs' paradox. Boltzmann resolved the paradox by introduc- 
ing the notion of indistinguishable particles and corrected for over counting of microstates by dividing Qby N\. 
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from which we get the ideal gas law: PV — NksT. This leads to 

dS^—dV^-PdV. (40) 

Consider a quasi-static process in which the system draws a quantity dQ of reversible heat and 
produces work equal to PdV. Thus dQ = PdV, from which it follows that dQ — TdS. Thus 
Boltzmann entropy is consistent with the thermodynamic entropy But Boltzmann liberated 
entropy from its thermal confines. We can now define entropy for a coin toss, S = ks log 2 or 
throw of a dice, S = ks log 6, etc. In general if an experiment has VL outcomes and they are all 
equally probable, then we can associate an entropy, ks log Ct, with the experiment. 



13 Boltzmann Entropy and Gibbs Entropy 

Consider an experiment of tossing N identical and fair coins. An outcome uj of this experiment 
is a string of Heads and Tails. We call u a microstate. The set of all possible microstates of 
the experiment is denoted by by Vl{N) called the sample space. The number of elements of the 
sample space is given by Vl{N) = 2^. Let us count the number of Heads in a string u and call it 
n{u;). The random variable n can take any value between and N. We call n a macro state. Let 
0.{n; N) — {u; : n(cj) = n} be the set of all strings having n Heads. In other words it is a set 
of all microstates belonging to the macro state n. The number of elements of the set il{n; N) or 
equivalently the number of microstates associated with the the macro state n, is given by 

ATI 

nin;N) = -— (41) 

n\{N — ny. 

We have 

N 

Q{N) = J2 ^(^; = 2^- (42) 

n=0 

^^The full expression for fl{E, V, N) obtained taking into account the momentum coordinates of the ideal gas 
molecules is given by the Sackur-Tetrode Equation, see below. 

Sackur-Tetrode Equation 

1 {2TTmEf^/^ 



n{E, V, N) 



f,3N jyl r(M + 1) ' 



where E is the total energy of the isolated system, ni is the mass of a molecule, h is Planck constant employed for 
coarse-graining the phase space {h^^ is the volume of a 6N dimensional cube in units of which the phase space volume 
is measured) and r(-) is the usual Ganoma function. 



/•oo 

r(n) ^ dt t"-^ e 
Jo 

The entropy of an ideal gas is thus given by. 



, 'E^/^V\ 5NkB SiVfcs, /47rm 
SiE, V, N) = NkB log —rjT + + lo, 



The above is known as Sackur-Tetrode equation. 
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Boltzmann associates an entropy S with each macro state n: 



Sin) = log 



n{n;N) 



(43) 



Note we have set the Boltzmann constant kB to unity. Boltzmann postulates that the system, switch- 
ing all the time from one microstate to another microstate, would evolve in an entropy increasing 
way and eventually reach an equilibrium state characterized by an unchanging value of n for which 
entropy is maximum. We immediately see that fl{n; N) is maximum when n — N/2. Therefore 
Boltzmann entropy for the equilibrium system is given by 



Sb = log 



n{n^N/2;N) 



= log 



L(f)' (f)! 



(44) 



Josiah Gibbs (1839 - 1903), proposed that equilibrium value of the macroscopic property n 
should be calculated by averaging over an appropriate ensemble of microstates. In the coin-tossing 
example considered here the ensemble consists of microstates from a Binomial distribution. Thus 
in Gibbs picture of statistical mechanics. 



N 



n, 



eq 



n=0 



2^ 2 

which is the same as that given by Boltzmann. Gibbs' entropy, given by 



(45) 



(46) 



is different from Boltzmann's entropy, see Eq. (44). However, in the thermodynamic limit, Gibbs 
entropy and Boltzmann entropy coincide. We have, in the limit of ^ oo. 



SG^SB + 0{l0gN). 



(47) 



14 Gibbs Ensembles 

Gibbs developed statistical mechanics into a fine tool for calculating equilibrium properties of 
macroscopic systems as averages over what we now call Gibbs' ensembles [31]. 

14.1 Microcanonical Ensemble 

The properties of an isolated system can be obtained by averaging over a microcanonical en- 
semble in which all microstates are of the same energy and occur with the same probability. For 
example footnote (25 ) expresses the number of microstates, flof N ideal gas molecules confined 
to a volume V and with energy E. 

14.2 Canonical Ensemble 

A closed system is one which exchanges only energy with the outside world and not material 
or volume. It is described by a canonical ensemble. The probability that a closed system will be 
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found in a microstate C, is given by 

P{C) = 



1 



exp 



ZiT,V,N) 

where P — 1/ {ksT) and Z{T, V.N) is called the canonical partition function given by 



(48) 



Z(r,y.Ar) = ^exp 



- PE{C) 



(49) 



Let Q{E, V, N)dE be the number of microstates of the closed system having energy between 
E and E + dE. We call Q,{E, V, N) the density of states. We can express the canonical partition 
function as. 



Z{T, V, AT) = j dE n{E, V, N) ex.p{-pE) 



(50) 



The density of states is a rapidly increasing function of energy: understandably so, since more 
the energy more is the number of ways of distributing it and hence more is the entropy. The 
exponential function decreases with increase of energy. The product of these two will be a sharply 
peaked function energy, peaking at the thermodynamic energy U — (E). A saddle point estimate 
of the integral can be made and we get. 



Z{T, V, N) = exp 



Kb 



(51) 



from which we get 

F(T, V, N) = -keT log Z{T, V, N) = U(S, V, N) - TS{U, V, N) (52) 

where F{T, V, N) is the Helmholtz free energy, proposed by Hermann von Helmholtz (1821-1894). 

14.3 Grand canonical ensemble 

An open system is one which exchanges both energy and material with the outside world. It is 
described by a grand canonical ensemble and the partition function is given by. 



Q{T,V,fi) = J2^xp 



I3E{C) + nl3N{C) 



(53) 



where N{C) is the number of molecules in the microstate C of the open system and n is the chemical 
potential. 

We can construct different Gibbs ensembles depending on the system we are investigating. 
Gibbs provided a general framework of statistical mechanics based on static Gibbs ensembles and 
averages over them. This is in contrast to the ensemble of Boltzmann which is dynamical. It is the 
typical behaviour that forms the basis of Boltzmann's picture of statistical mechanics. 
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The expression for entropy given by Eq. (37) was also derived by Claude Elwood Shannon 
(1916 - 2001), in the context of information theory [32]. We say Eq. (37) defines Boltzmann- 
Gibbs-Shannon entropy 

Boltzmann entropy, laid the foundation for statistical mechanics - a subject that helps us cal- 
culate macroscopic properties of an equilibrium system from those of its microscopic constituents 
and their interactions. This subject has since grown to a very high degree of sophistication. More 
importantly the predictions of statistical mechanics have been borne out by experiments. Statistical 
mechanics has become such a successful enterprise that physicists do not anymore question the use 
of statistics for describing macroscopic phenomena But the nagging doubt remains: What is the 
origin for the observed stochasticity ? 



15 Dynamical Entropy from Chaos 

Then came a meteorologist and mathematician named Edward Norton Lorenz with his three cou- 
pled first-order nonlinear differential equations. He had obtained them by truncating Navier-Stokes 
equations The three equations of Lorenz were intended to provide a simple and approximate 
description of atmospheric behaviour. Lorenz was solving them on a computer. He discovered [40] 
that he had two very different numerical solutions for the same problem with almost identical initial 
conditions. This chance observation heralded a new field called chaotic dynamics [41,42]. Two 
phase space trajectories of a chaotic system starting off from arbitrarily close phase space points 
diverge exponentially and become completely uncorrected asymptotically. This means that you 
can not have any hope of making any long term predictions from deterministic equations if they 
happen to be chaotic. In other words determinism does not necessarily imply predictability. 

Possibility of dynamical instability due to sensitive dependence on initial conditions was known 
to Poincare [28, 29]. I have already mentioned of this while discussing Loschmidt's reversibility 
paradox [26] and Zermelo's recurrence paradox [27]. But the full import of Poincare 's findings 
was lost on the physicists for over half a century. They did not think much of it until computers 
arrived on their desktops and helped them see on graphic terminals, the strange trajectories traced 
by chaotic dynamical systems. 

A standard way of determining whether or not a dynamical system is chaotic is to calculate 
the Lyapunov exponent. There are as many Lyapunov exponents as the dimensions of the phase 
space. Consider dynamics in an n dimensional phase space. Consider an n - dimensional sphere 
of initial conditions. At a later time all the trajectories emanating from the sphere will form an n- 
dimensional ellipsoid. We can calculate a Lyapunov exponent for each dimension. When talking of 
a single Lyapunov exponent we normally refer to the largest and denote it by A. Thus if A > 0, we 

^^Ever since, there have been several 'entropies' proposed in different contexts. These include for example, Fisher 
information [33], von Neumann entropy [34] Renyi entropy [35], Kolmogrov- Sinai entropy [36], Algorithmic entropy 
[37], Tsallis entropy [38] and Kaniadakis entropy [39|. 

^^Physicists were in for a greater embarrassment with the advent of quantum mechanics. Statistics enters into 
microscopic laws. Stochasticity is intrinsic to quantum mechanics. The notion of ensemble of Maxwell, Boltzmann 
and Gibbs came in handy in describing the results of measurements in quantum mechanics. 

^'^The Navier-Stokes equation of Claude Louis Marie Navier (1785-1836) and Sir George Gabriel Stokes (1819- 
1903), is the primary equation of computational fluid dynamics, relating pressure and external forces acting on a fluid 
to the response of the fluid flow. Forms of this equation are used in computations for aircraft and ship design, weather 
prediction, and chmate modehng. 
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say the dynamics is chaotic. The system becomes unpredictable for times greater than of the order 
of 1/A. On these asymptotic time scales the system becomes amenable to statistical description. 

We recognize thus, that at least in principle, Chaos provides raisin d'etre for statistics in statisti- 
cal mechanics. All systems that obey the laws of thermodynamics are chaotic. Nonlinear dynamics 
and chaos provide the link between deterministic micro and the stochastic macro - a link that Boltz- 
mann was struggling to figure out. 

In fact Boltzmann's interpretation of probability was entirely based on dynamics. The dynam- 
ical trajectory of an isolated equilibrium system is confined to a constant energy surface in a 6A^ 
dimensional phase space. Boltzmann first shows that the phase space density p remains constant 
along a trajectory; this is now called the Liouville theorem. He then assumes that all the points on 
the energy surface lie on a single trajectory. This is called ergodicity. Then p{x) = 5{H{x) — E) 
is the stationary density, where Ti. is the Hamiltonian, E is energy and 5 is the usual Dirac delta 
function. 

Boltzmann's ergodicity has been generalized by Sinai [43], Ruelle [44] and Bowen [45] to 
describe dissipative systems in a steady state. The strange attractor of the dissipative dynamics is 
the non-equilibrium analogue of the equilibrium constant energy surface considered by Boltzmann. 
The SRB measure [44, 46] on the attractor expressed in terms of phase space volume contraction 
is analogous to the Liouville measure on the energy surface of an equilibrium isolated system. 
Such a generalization permits assignment of dynamical weights to non-equilibrium states. These 
weights, let me repeat, are based on the dynamical properties of the microscopic constituents of a 
macroscopic system. To appreciate the import of this statement, we must recognize that words like 
equilibrium, heat, entropy, temperature etc., belong to the vocabulary of the macroscopic world of 
thermodynamics. They do not have any meaning in the microscopic world . Paraphrasing Maxwell, 
at microscopic level you can not tell heat from work, since both are essentially energy, see below. 

15.1 Microscopic description of work and heat 

Let U denote the thermodynamic energy of a closed system obtained by averaging the statistical 
mechanical energy E, over a canonical ensemble of microstates. Let the microstates accessible to 
the system be indexed by natural numbers i. Let pi denote the probability for the system to be in 
microstate i whose energy is Ei . The thermodynamic energy U is then given by. 



We have formally. 



dU 





(54) 



Thus we can change the energy of a system by an amount dU, through work dW, given by the first 
term on the right and/or heat grcv, given by the second term on the right in the above equation. 
Thus work and heat are simply two modes of energy transfer. 
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Work 

To identify the first term as work we proceed as follows. We have. 



d_ 
dV 



-PdV 



dw , 



(55) 



where P denotes pressure. Work corresponds to change in energy of the macroscopic system 
brought about by changing the energies of its microstates without altering in any way their proba- 
bilities {pi}. 

Heat 

To identify the second term as heat, we start with the definition of Boltzmann-Gibbs-Shanon en- 
tropy 



and proceed as follows. 



dqrey = TdS 

= -ksT Y dpi - ksT ^ logpidpi 

i 

= -ksT Y dpi - ksT ^ logpidpi 

i 

= -kBT^dpilogPi 



= keTj^dPi 

i 

= dpi , 



pEi + logZ 



(56) 



where Z denotes the canonical partition function, see Eq. (49) . In the above derivation we have 
made use of the fact that in a canonical ensemble describing a closed system, pi = exp{—PEi). 
Thus heat is change of energy of a closed system brought about by changing the probabilities {pi} 
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without altering in any way the energies {Ei} of the microstates. It is in fact because of this 
identification we relate heat and hence entropy to randomness. 

In the phase space of the thermodynamic variables only an equilibrium system can be repre- 
sented by a point; only a quasi-static process can be represented by a curve. However in the 6A^ 
dimensional phase space of statistical mechanics, a macroscopic system in equilibrium or not, can 
be represented by a point; any process can be represented by a trajectory, to which we can attach a 
suitably defined dynamical weight. 

Thus dynamical measures of recent times, have liberated the notion of entropy from its equi- 
librium and quasi-static confines, into non-equilibrium realms. We have, indeed, come a long way: 
from the thermal entropy of Clausius to the statistical entropy of Boltzmann (both applicable to 
equilibrium systems and quasi-static processes), and now to the SRB measures (defined for non- 
equilibrium systems and processes). Recently SRB measure has been shown to provide a correct 
description [47] of a far from equilibrium system in a computer simulation [48]. 



16 Entropy Fluctuation Theorems 

These new developments are embodied in what we call fluctuation theorems [48,49]. The general 
idea behind a fluctuation theorem can be stated as follows. Let denote entropy production 
rate calculated by averaging over segments of a long trajectory of duration r. Note that S^- is a 
dynamical entropy obtained from observing the phase space expansion/contraction. Let n(S'T-) be 
the probability of Sr. This can be calculated by considering an ensemble of long trajectories each 
of duration r. Fluctuation theorem states, 

^''^^^ -e^pirSr]. (57) 



U{-Sr) 

Fluctuation theorem helps us calculate the probability for the entropy to change in a way opposite 
to that dictated by the Second law; this probability of Second law violation is exponentially small 
for large systems and for long observation times. By the same token fluctuation theorems predict 
and more importantly quantify Second law violation in small systems and on small time scales of 
observation. The predictions of fluctuation theorems have since been verified experimentally [50, 
51]. See also [52] for an interesting examination of the experimental tools of fluctuation theorems. 



17 Jarzynski Identity 

In the year 1997, C. Jarzynski [53] discovered a remarkable identity relating non-equilibrium work 
fluctuation to equilibrium free energies. Consider a switching process, discussed earlier, carried 
out over a time r, with the system thermostatted at temperature T — l/(kBP). Let W denote 
the work done during the switching process. We carry out the switching several times and collect 
an ensemble {Wj}, formally represented by the probability density p{W;t). All the switching 
experiments are carried out with the same protocol. If r = oo, the process is quasi-static. We have 
Wi = Wr V i. The work done is called reversible work, Wr. For a general switching experiment 



^^A thermostat exchanges energy with the system without changing its temperature or performing any work 
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where r < oo, the Second law says that 



AF < (W), 

where AF is the change in the Helmholtz free energy 
Jarzynski's identity is given by. 



exp{-pW)^ = exp (-/3AF) , 



where (•) denote averaging over the distribution of W. 
17.1 Jarzynski identity and the Second law 

It may be noticed that since the exponential function is convex, we have. 



(58) 



(59) 



eicp{-pw) ) > exp 



P{W) 



(60) 



which in conjunction with Jarzynski's identity implies that. 



exp(-/3AF) > exp 



-P{W) 



-pAF > -p{W), 



AF < (W), 



(61) 



which is a statement of the Second law. In this sense, proof of Jarzynski's identity is a proof of the 
Second law. 

17.2 Jarzynski identity: cumulant expansion 

Let us express Jarzynski's equality as a cumulant expansion [54], 



exp(-/3Vr)^ = exp 



.n=l 



= exp(-/3AF) , 



(62) 



where Cn denotes the n— th cumulant of W. The cumulants and the moments are related to each 
other. The n-th cumulant can be expressed in terms of the moments of order n and less. The 
first cumulant, Ci is the same as the first moment (W); the second cumulant, ^2 is the variance 
cr2 = (^W'^) — {W)'^; etc. From the cumulant expansion given by Eq. (62), we get. 



n=3 



(63) 
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17.2.1 Reversible work and free energy 

Consider a quasi-static switching process for which, 

p {W;r ^ oo) ^ S {W - Wr) , 



(64) 



by definition. Then, in Eq. (63), only the first term (of the cumulant expansion) is non-zero. We 
get. 



(65) 



consistent with thermodynamics. 



17.2.2 Fluctuation and dissipation 

Now consider a switching process, during which the system remains very close to equilibrium; it is 
reasonable to expect the statistics of W to obey the Central Limit Theorem Hence p{W; r > > 0) 
shall be a Gaussian; for a Gaussian, all the cumulants from the third up-wards are identically zero; 
hence, in Eq. (63), only the first two terms survive and we get 



AF={W)--/3a'^ 



(66) 



Dissipation given by. 



(67) 



^"According to the Central Limit Theorem, the sum of N independent and identically distributed, finite variance 

random variables, has an asymptotic (TV oo) Gaussian distribution with both mean and variance diverging linearly 
with N. This means that the relative fluctuation is inversely proportional to Vn and hence is small for large N. See 
e.g. [55]. This is easily seen as follows. 

The Central Limit Theorem 



Let 



where {X^ : i = 1, N} are identically distributed independent random variables with zero mean and finite variance cr^. 
Let $x {k) denote the characteristic function of the random variable X. Then 



N 



N 



exp 



jv^oo exp 



n=3 



where Cn denotes the n— th cumulant of X. The Fourier inverse of the asymptotic (A'' — > oo) expression for $r (fc) is 
Gaussian with mean zero and variance a^. 
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is proportional to fluctuation, a^r. This result is identical to the fluctuation dissipation relation 
of Callen and Welton [56]. See [57] for an interesting discussion on Gaussian Work fluctuation, 
Jarzynski identity and fluctuation dissipation theorem. However, if the switching process drives the 
system far from equilibrium, the work distribution would no longer be Gaussian and we need to 
include contributions from higher order cumulants to calculate the dissipation (Wd) and hence free 
energy: AF — (W) — (Wd). Jarzynski's equality has been shown to hold good for Hamiltonian 
evolution [53] as well as stochastic evolution [58]; its validity has been established in computer 
simulation [58] and in experiments [59]. 

18 Microscopic Reversibility and Crooks identity 

In another parallel, independent and interesting development, Gavin E. Crooks [60] discovered a 
fluctuation theorem for a thermostatted, Markovian dynamical process. During the process, the 
degree of freedom A switches from an initial value of Aq to a final value An in N time steps. The 
switching process is not necessarily quasi-static. 

18.1 Heat Step, Work step and Markov Chain 

The system is initially in a microstate Co(Ao) G f2(Ao), where 0(Ao) denote the set of all 
microstates of the system with A = Aq. Each step is considered as made up of a heat sub-step: 
Co(Ao) — > Ci(Ao) and a work sub-step: Ci(Ao) Ci(Ai). Thus we get a Markov chain of 
microstates given by, 

.F|Co(Ao) = Co(Ao) ^ Ci(Ao) ^ Ci(Ai) ^ • • • ^ Cfc(Afe) ^ G^+iiAk) ^ Cfc+i(Afc+i) ^ 

• • • ^ CAr_i(A7v_i) Cn{An-i) Cn{An) 

Let us consider a heat sub-step Cfc(Afe) — > Cfc+i(Afc), described by a Markov transition matrix 
M{k) whose elements are given by 

M,,,(A;) = P(Cfc+i = C,\Ck = C,), (68) 

where Ci E Cl{Ak). We have used script symbol C to denote microstates of the system and roman 
symbol to denote those on the Markov chain with k serving as the time index. The matrix M(k) 
has the following properties: 

• The elements of M{k) are all non-negative: 

Mij{k)>Oyt,j. 
Note Mij denotes (transition) probability. 

• M{k) is column stochastic: 

5]M,,(A;) = lVj. 

i 

This follows from the normalization. After a step the system must be found in any one of its 
microstates with unit probability. 

^'A Markov chain describes the time evolution of a system with a finite or countable number of microstates. We 
also specialize to Markov chain in discrete time. In general for a Markov process, the past has no influence over the 
future once the present is specified. 
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M{k) is regular: There exists an integer n > 0, such that 

([M{k)r) >ovi,i. 

This ensures ergodicity. 

M{k) is balanced: There exists a unique invariant probability vector |vr(A;)) such that 

M{k)\7r{k)) = \7r{k)). 

|7r(A;)) describes the equilibrium distribution of the closed system at (3 and with A = Ak. The 
components of |7r) are given by. 



exp 

TTiik) = 



Z{(3,Ak) 



(69) 



where E{Ci, Ak) is the energy of the microstate Ci belonging to the system with A = A^. The 
canonical partition function is denoted by Ak). 

18.2 Metropolis and Heat-bath algorithms 

We need a model for M{k). For example Metropolis algorithm [62] prescribes, 

^i,j{k) = a X min [ 1, ) and i 7^ j, (70) 

M,,,{k) = l-^M,-,Vz, (71) 

where the constant a has been introduced to ensure that no diagonal element is negative or exceeds 
unity. 

The heat-bath algorithm [63] also known as Glauber algorithm [65] or Gibbs' sampler [64] is 
given by 

M,,, = ^^ V (72) 

TTi + TTj 

Once a model for M is defined, we can calculate the probability for the Markov chain Co(Ao), 
where we take each work sub- step with unit probability. 



^^Peron-Frobenius theorems, see e.g. [61], tell us the following. The largest eigenvalue of M is real and non 
degenerate. Its value is unity. All other eigenvalues of M are much less than unity in modulus. The right eigenvector 
associated with the largest eigenvalue is called the invariant or equilibrium probability vector and is denoted by \tt). 
The eigenvectors of M are linearly independent and span the vector space of M. We have M"|(/)) ^ |7r), for n ^ 00 
and for {(j)\'K) ^ 0. Physically it means that the system eventually relaxes to its equilibrium state starting from any 
arbitrary non-equilibrium state. Also once the system reaches equilibrium it continues to be in equilibrium. Further 
action of M does not change its state. M\-k) = 
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18.3 Time-Reversal of Markov Chain 

Let us now run the Markov chain back- wards and call it the time - reversal. Let 7^|Civ(Ajv) 
denote the time reversal of J^|Co(Ao). It is given by, 

7e|Cjv(Aiv) = Cn{An)^Cn{An-i)-^Cn-i{An-i)^--- 

^ Cfc+i(Afe+i) ^ C,+i(A,) ^ C,(A,) • • • ^ Ci(Ai) ^ Ci(Ao) ^ Co(Ao) 

Note that in the time-reversed Markov chain, the work sub-step comes first followed by the heat 
sub-step in every time step. We need to calculate the probability for the time reversed Markov 
chain. Reversing the work sub-step is easily visualized. We switch the parameter A back-wards 
with unit probability. Let the time reversed heat step Ck+i{Ak) — * Cfc(Afc) be described by M{k), 
called the reversal of M{k). To construct M{k) we proceed as follows. 

Consider the heat sub-step Cjt+i(Afe) — > Cfe(Afe) in the forward Markov chain. Define a two- 
step joint probability matrix W whose elements are given by. 



= Mi,,{k)7i,{k) (73) 

In the above the second step follows from the definition of conditional probability. Thus, given 
M{k) we can get the corresponding W{k) and vice versa. To this end we define a diagonal matrix 
D(k) with elements, 

D,j{k)^7ri{k)Sij. (74) 

Then, 

W{k) = M{k)D{k) (75) 

and 

M(k) = W{k)D-\k) (76) 
Also it is easily checked that is matrix-stochastic: 

i j 

Let W{k) denote the time reversal of W{k). A little thought will convince you that a good choice 
of W{k) is W\k), where the superscript ^ denotes transpose operation. The corresponding M{k) 
can be obtained as follows. 

M{k) = W{k)D-\k) 
= W\k)D-\k) 



= D{k)M\k)D-\k). 



(78) 
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We say a Markov chain is time symmetric if W is symmetric. In other words, 

W{k) = W\k) = W{k), (79) 
for time symmetry. Also for a time symmetric Markov chain, we have 

M(k) = W{k)D-\k) 

= W'{k)D-\k) 

= W{k)D-\k) 

= M(k). (80) 



This implies that TTj{k)Mij{k) = 7ri{k)Mj^i{k), called detailed balance A sequence of mi- 
crostates visited by an equilibrium system constitutes a time-symmetric Markov chain. 

18.4 Crooks Identity 

The probability of TI\Cn{An) can be calculated from the matrices {M{k)}. Let Iljr denote 
the probability of jF|Co(Ao) and IIt^ that of its reverse. The ratio of these two probabilities can be 
calculated and is given by. 



— = exp 



(81) 



where Q is the energy absorbed by the system, in the form of heat, from the thermostat during 
forward Markov chain evolution. The above is called Crooks identity. The import of Crooks' 
finding can be understood if we consider switching from an equilibrium ensemble at {3 and with 
A = Ao to another equilibrium ensemble at the same (3 but with A = Ajv through a process which 
is not necessarily quasi-static. Thus Co(Ao) and Cn{An) belong to equilibrium ensembles at the 



^^The Metropolis [62] and the heat-bath algorithms [63-65] obey detailed balance. There are algorithms that do not 
obey detailed balance. It is often said that a simple balance condition, A'/|7r) = |7r) is adequate to drive the system to 
equilibrium in a computer simulation, see e.g. [66, 67]. We see that it is detailed balance that ensures time symmetry in 
a sequence of microstates visited by the system after equilibration. If the computer algorithm obeys only balance and 
not detailed balance then time asymmetry in the Markov chain of microstates sampled, would be present even during 
equilibrium runs. 
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same temperature. Then 

nfco(Ao 



n c^(Aiv; 



X 



exp 




Z{(3,An) 






exp 


- (3E (cn{An)^ 



X exp(-/3(5) 



exp 



(3lAE - AF -Q 



exp 



pi {W) - AF 



— exp 



(82) 



In the above we have used the definition of free energy, F(T, V, N) = —ksT log Z(T, V, N) for 
going from the first Une to the second line. In going to the third line from the second, we have made 
use of the first law: (1^) = AE — Q. Physically Eq. (82) means that the probability of finding a 
dissipating segment of a Markov chain evolution is exponentially large compared to that of finding 
its reverse. Starting from Crooks identity we can derive fluctuation theorems and Jarzynski's equal- 
ity, see e.g. [68]. Very recently Cleuren, Van den Broeck and Kawai [69] have derived equivalent of 
Crooks identity in microcanonical ensemble description and have obtained analytical expressions 
for the work fluctuations in an idealized experiment consisting of a convex body moving at constant 
speed through an ideal gas. Crooks identity has since been verified experimentally [70]. 



19 Epilogue 



Thus, recent developments have helped improve our understanding of the issues that link time 
asymmetric macroscopic world to the time symmetric microscopic world. These developments 
are not inconsistent with the hunch Boltzmann had. Let me conclude a la Cohen [71], quoting 
from Boltzmann. In his 1899 lecture at Munich, Germany, on recent developments of methods 
of theoretical physics [72], Boltzmann talks of the conflict between dynamics and statistics in 
describing macroscopic phenomena. He asks if statistics would continue to dominate in the future, 
or would it give way to dynamics. He concludes saying ' • • • interesting questions! One almost 
regrets to have to die long before they are settled. Oh! immodest mortal ! Your destiny is the joy of 
watching the ever-shifting battle' . 
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